MEMORANDUM November 15, 2019 


TO: Jessica Chevalier 
Secondary Literacy Director 


FROM: Carla Stevens 
Assistant Superintendent, Research and Accountability 


SUBJECT: THE LITERACY EMPOWERED INITIATIVE: EFFECT ON STUDENT 
PERFORMANCE IN HISD CORE COURSES, 2018-2019 


Literacy Empowered (LE) was initiated during the 2017-2018 school year to provide literacy 
support for all Houston Independent School District (HISD) high school students through the 
reading of self-selected and assigned texts, the proficient use of PowerUp tools, writing, 
discourse, and authentic literacy practices across all foundation courses. This evaluation 
reported on teacher participation in the LE professional development, and the effects of the LE 
initiative on student performance in core high school courses using the 2019 State of Texas 
Assessments of Academic Readiness (STAAR) End-of-Course (EOC) exam results. The 
evaluation was guided by four questions and used descriptive statistics, treatment effects, and 
multiple regression analyses to report on the effects of the LE initiative. 


Key findings include: 


e Most teacher participants (78.7%) completed three or six PD credit hours for LE and were 
not subject to the research-recommended yearlong support required for effective 
professional development. 

e A higher percentage of students in the treatment group compared to the non-treatment 
group were male, black or Hispanic, economically disadvantaged, at-risk for school dropout, 
enrolled in special education, had limited English proficiency (LEP), and came from homes 
where Spanish was the predominant language spoken. 

e Students whose teachers did not participate in the LE Professional development 
outperformed students on the 2019 STAAR Algebra I, English |, English Il, and U.S. History 
EOC tests whose teachers participated in the PD. The difference was statistically significant 
(p<.05; p<.001) and ranged between 41.6 and 92.5 scale score point (ssp). 

e A higher percentage of students whose teachers did not participate in the LE professional 
development performed at the Approaches, Meets, and Masters Grade Level Student 
standards on the 2019 STAAR Algebra |, Biology, English |, English Il, and U.S. History 
EOC exams compared to students whose teachers participated in the PD. 

e When disaggregated by student groups in the study, generally, a higher percentage of 
students whose teachers did not complete the LE professional development, with some 
exceptions, performed at or above the Approaches Student Standard on the five STAAR 
EOC tests when compared to their peers whose teachers participated in the PD. 


Further distribution of this report is at your discretion. Should you have any questions, please 
contact me at 713-556-6700. 


CJS 
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Literacy Empowered Initiative: Effect on Student 
Performance in HISD Core Courses, 2018-2019 


Executive Summary 


Literacy Empowered (LE) is an initiative designed to improve literacy across the curriculum for high school 
students in the Houston Independent School District (HISD). Literacy Empowered is an extension of the 
Literacy by 3 and Literacy in the Middle initiatives designed to ensure that all students can read on grade 
level as stipulated in HISD performance goals. First implemented during the 2017-2018 school year, the 
initiative targets ninth through twelfth-grade students and focuses on reading, writing, discourse, and the 
use of PowerUp tools. It facilitates students’ engagement in reading self-selected and assigned texts at 
students’ reading levels to expand their repertoire of strategies that meet the demands of increasingly 
complex texts. Students also engage in authentic writing in each discipline to extend and provide evidence 
of their understanding. Students also engage in daily academic discourse involving grade-level 
personalized texts, inquiry, and authentic writing opportunities. 


The purpose of this evaluation was to determine the effect of the LE initiative on the academic performance 
of students on the 2019 State of Texas Assessments of Academic Readiness (STAAR) End-of-Course 
(EOC) tests. Students whose teachers completed the professional development (PD) for LE constituted the 
treatment groups and those whose teachers did not participate in the LE professional development were 
the non-treatment group. Treatment effects with regression adjustment (teffects ra) was used to determine 
the initiatives’ effects. Students’ performance by treatment levels and disaggregated by EOC tests and 
student groups were also presented in the study. The study also analyzed teachers’ participation in the 
Literacy Empowered PD. 


Key Findings 


e Most LE teacher participants (78.7%) completed three or six credit hours of PD and were not subject 
to the yearlong support required for research-based effective professional development. 


e A higher percentage of students in the treatment group compared to students in the non-treatment 
group were male, black or Hispanic, economically disadvantaged, at-risk for school dropout, were 
enrolled in special education, had limited English proficiency (LEP), and came from homes where 
Spanish was the predominant language spoken. 


e Students whose teachers did not participate in the Literacy Empowered PD outperformed students on 
the 2019 STAAR Algebra |, English I, English Il, and U.S. History EOC tests whose teachers 
participated in the PD. The difference was statistically significant (p<.05; p<.001) and ranged between 
41.6 and 92.5 scale score points (ssp). There were no statistically significant between-group differences 
for the Biology EOC test. 


e A higher percentage of students whose teachers did not participate in the Literacy Empowered PD, 
compared to students whose teachers participated in the PD, met the Approaches, Meets, and Masters 
Grade Level Student standards on the 2019 STAAR Algebra |, Biology, English |, English II, and U.S. 
History EOC exams compared to students whose teachers participated in the PD. 


e When disaggregated by student groups in the study, generally, a higher percentage of students, whose 
teachers did not complete the LE professional development performed at or above the Approaches 
Grade Level Student standard on the five STAAR EOC tests when compared to their peers whose 
teachers participated in the PD. There were exceptions for some groups in different content areas. 
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Recommendations 


Since most LE teachers in the study completed three or six credit hours of professional development, 
future LE professional development should be designed to reflect U.S. Department of Education 
Institute of Education Sciences What Works Clearinghouse research on professional development 
which found that at least 14 to 49 hours of initial professional development and yearlong support for 
teachers during program implementation had statistically significant effects on students’ achievement. 


Because being identified as at-risk for school dropout was the strongest but an inverse predictor of 


performance on the 2019 STAAR EOC tests, it may be essential to address student issues associated 
with school dropout to improve the effectiveness of the LE initiative. 
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Introduction 


In 2016-2017, the Houston Independent School District (HISD) began the implementation of early literacy 
initiatives to support the district's goal “to have all students reading on or above grade level by the end of 
third grade” (HISD, 2018, p. VIII-54). Literacy Empowered combined with the PowerUp program is an 
extension of this goal into high school to “ensure that all high school students can engage in personalized 
learning experiences and become global graduates” (HISD, 2018, p. VII-54). Students are expected to read 
authentic, self-selected, and assigned texts, daily, for at least 120 minutes within their four core foundation 
classes. 


Literacy Empowered harnesses technology-based literacy instruction to transform learning so that students 
can attain the knowledge and skills essential for college and their careers. Students receive “standards- 
based models of thinking and learning in whole-group, small-group, or conference settings based on need 
and have time to read, write, and discuss authentically at their own instructional levels” (HISD, 2018, p. VIII- 
54). Literacy Empowered targets ninth- through twelfth-grade students in reading, writing, and discourse to 
expand their system of strategic actions that meet the demands of increasingly complex texts. Students 
also engage in authentic writing tasks in each discipline to extend and provide evidence of their 
understanding. Finally, students engage in daily academic discourse around personalized grade-level texts, 
inquiry, and authentic writing opportunities. 


Literacy Empowered is based on the premise that the transformation of Tier | instruction among HISD high 
school leaders and teachers requires the tools and pedagogies that support 21S-century students. The 
deployment of a 1:1 computer device and the creation of core content-areas master courses ensure that 
students have the tools necessary to personalize their learning paths in accordance with state standards 
(HISD, 2018). “Just in time” support and the use of literacy principles provide support for learners in 
strengthening their literacy skills and to close the literacy gaps (HISD, 2018, p. VIII-55). Further, support for 
students and teachers include the procurement of materials and resources that include (1) classroom 
libraries with a wide range of Lexile level texts, (2) digital libraries with personalized texts in all four core 
foundation disciplines, (3) comprehensive toolkits for large and small-group lessons, and (4) DBQ? kits for 
social studies classes to support comprehension, analyses, and writing (HISD, 2018, p. VIII-55). Schools 
were expected to maintain and increase their existing campus resources. 


The purpose of this evaluation was to determine the effect of Literacy Empowered on the academic 
performance of HISD students during the 2018-2019 school year. The evaluation was guided by the 
following questions: 


1. What was the PD participation for Literacy Empowered among teachers in the study? 
2. What was the demographic and educational composition of students in the LE study sample? 


3. How did students, expose to LE, perform on the 2019 State of Texas Assessments of Academic 
Readiness (STAAR) End-of-Course (EOC)? 


4. Towhat extent did LE contribute to the 2018-2019 reading achievement of high school students in 
the sample? 


Literature Review 


Since the 1970s, secondary schools’ reading scores have remained flat due, in part, to relatively little 
investment in literacy (Heller & Greenleaf, 2007). The National Assessment of Education Progress (NAEP) 


1 A document-based question (DBQ), also known as data-based question, is an essay or series of short-answer questions 
that is constructed by students using one's own knowledge combined with support from several provided sources. Usually it 
is employed on timed history tests. 
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twelfth-grade results for 2015 indicated that, nationally, only 37 percent of students were reading at or 
above the Proficient? level (IES, 2015). Without effective and ongoing literacy instruction, students who are 
behind in reading when they enter middle grades, likely, will never catch up (IES, 2017, p. 2). Data from the 
2017 National Assessment of Educational Progress (NAEP) Reading Trial Urban District Snapshot Report 
revealed that 18 percent of HISD eighth-grade students were reading at or above the Proficient level and 
59 percent at the Basic level. Neither performances were significantly different than they were in 2002 (17% 
and 59%, respectively), although the cohorts were different (IES, 2017). 


Data from the 2018 HISD STAAR EOC spring report showed that from 2015 through to 2018 between 57 
and 64 percent of first-time testers met the Approaches Student Standard on the English | EOC 
assessments, 40 and 48 percent met Meets Student Standard, and between 8 and 9 percent met Masters 
Student Standard (HISD Research & Accountability, 2018a), respectively. When disaggregated, the 
reading performance for Black, Hispanic, or economically disadvantaged student groups revealed greater 
disparities when compared to their Asian, White, or non-economically disadvantaged peers. 


Research identified several effective strategies for high school literacy instruction (Heller & Greenleaf, 
2007). These included assessment of students’ reading on high school entry to identify reading needs and 
to intervene, (2) supporting low-level readers and helping them to make progress in reading fluency, basic 
comprehension, and other skills, (3) making special effort to motivate students, many of whom have been 
demoralized by years of reading failure, and (4) engaging student in reading and writing assignments that 
tap into their areas of interest (Heller & Greenleaf, 2007). The National Council of Teachers of English 
(2018) position statement on adolescent literacy instruction identified several strategies for teaching 
reading. These included modeling how students access specific content-area text and using conversations 
and discussions regarding texts that are authentic, student-initiated, and teacher-facilitated. It also included 
the use of diverse text interpretations supported by textual evidence that deepens the conversations and 
discussions and acknowledges and considers the cultural frameworks that influence reading and the 
application of metacognitive strategies (National Council of Teachers of English, 2018). 


Deshler and Hock (2007) proposed a theory of adolescent reading that hinges on word recognition 
(decoding, accurate sight-word recognition, fluent word reading, and so on) and language comprehension 
(factual and conceptual knowledge, vocabulary, language and text structure, and so on). Strategies 
included explicit instruction and scaffolding to support struggling readers with word-level intervention and 
explicit instructions in language comprehension and reasoning that integrated cognitive and metacognitive 
strategies (Deshler & Hock, 2007). 


Previous evaluation of LE in HISD found that the differences in performance between LE and non-LE 
students ranged from 23.0 to 133.9 scale score points (ssp) on the 2018 STAAR EOC assessments, in 
favor of non-LE students, that is, students whose teachers did not participate in the LE professional 
development (HISD Research & Accountability, 2018b). The evaluation also involved a survey of LE 
teachers. Teachers surveyed gave relatively high ratings for their use frequency of LE instructional 
strategies and practices (4.0 and above of 5.0) but gave lower ratings (2.29 to 3.14 of 5.0) for their use of 
the universal screener to assess students’ reading performance and growth and to place them into flexible 
reading groups (HISD Research & Accountability, 2018b). Assessing students periodically and using the 
assessment results for organizing them into flexible reading groups were critical components for the 
implementation fidelity of LE. Being at risk for school dropout, special education, and being identified as 
Gifted and Talented (G/T) were among the strongest predictors of performance among students in the 2018 
LE sample. G/T was a positive predictor (HISD Research & Accountability, 2018b). 


Overall, the literature on adolescent literacy advocates for explicit literacy instruction on word recognition 
for struggling readers and similarly, explicit instruction on language comprehension using cognitive and 
metacognitive strategies, and sustained reading and writing, conversations, and discussion on authentic, 


2Students performing at or above the Proficient level on NAEP assessments demonstrate solid academic performance 
and competency over challenging subject matter (NCES, 2012). 
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student self-selected texts with teachers operating as facilitators. These strategies appear to be consistent 
with LE instructional approaches that include authentic, self-selected texts, daily academic discourse 
around grade level and personalized texts, and authentic writing opportunities. Consistent with the literature 
(Deshler & Hock, 2007; National Council of Teachers of English, 2018), LE also advocates for the use of 
digital texts. It can be concluded that if implemented with fidelity, LE should result in improved performance 
of students whose teachers underwent professional development to deliver the initiative. 


Method 


This is a quasi-experimental study designed to measure the effect of LE on the academic performance of 
students during the 2018-2019 school year. Students whose teachers completed the LE professional 
development made up the treatment group and students whose teachers did not participate in the PD 
formed the non-treatment group. Only first-time testers on the STAAR regular exams with a legitimate score 
were included in the sample. First-time testers most likely had a single year exposure to LE. 


Data collection 


Teacher PD data for this evaluation were gathered from the HISD eLearning database that archives the PD 
of HISD teachers. High school teachers of core or foundational courses who completed the LE summer 
professional development were selected from the database and used in the evaluation. A total of 94 
teachers in the core subject areas completed face-to-face or online course versions of LE professional 
development in August, September, and November 2018. Professional effectiveness research found that 
a minimum of 14 hours of professional development had a statistically significant effect on student 
achievement. Moreover, an average of 49 hours of professional development was found to increase student 
achievement by 21 percentile points. Initial professional development must be followed by yearlong support 
(Yoon, Duncan, Lee, Scarloss, & Shapley, 2009). 


Teacher data were linked to student educational and demographic variables downloaded from Chancery 
Ad Hoc via Cognos? using unique identifiers. These variables included students’ race and ethnicity, gender, 
special education status, limited English proficiency, and economic status, at-risk and gifted and talented 
statuses. These, in turn, were linked to students’ STAAR data using these identifiers. STAAR Algebra |, 
English I, English Il, Biology, and U.S. History EOC exam data were used as the outcome measure. The 
STAAR data included students’ scale scores and their performance on the state standards. Overall, 53,610 
students made up the study sample. Of these, 3,329 represented students whose teachers completed the 
PD constituted the treatment group and 50,282 constituted the non-treatment group (the group of students 
whose teacher did not participate in the PD). 


Data Analysis 


The data was cleaned and LE and non-LE students identified and coded as treatment and non-treatment 
groups, respectively. The data met homoscedasticity, normality, and linearity conditions. The data were 
subjected to treatment effects with regression adjustment (teffects ra) to estimate the effect of LE on 
students’ performance on the 2019 STAAR EOC tests. Teffects ra is a Stata command. Stata is a statistical 
analysis software. It runs separate regressions for each treatment level, selects matching treatment and 
non-treatment groups and run the effects in one operation. It calculates the predicted means, that is, the 
potential outcome means, (POM) of the exam results for each EOC by treatment level (LE and non-LE 
students) and uses the differences in the means as estimates of the program effects. The average treatment 
effect (ATE) is used in this study. It is the average performance of any student selected at random from the 
sample if those students were exposed to the LE initiative. Teffects were regressed using key demographic 
and educational variables, including gender, ethnicity, limited English proficiency (LEP), special education, 


3 Cognos is an International Business Machine (IBM) Corporation data querying software used to download data from the HISD 
Chancery ad hoc data warehouse. 
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gifted and talented, at-risk, career and technical education, and economic status. The results of the analysis 
are presented in tables and graphs. 


The percentages of students in the sample who met the Approaches, Meet, and Masters Grade Level 
Student standards by EOC test were also presented. Students who met the Approaches Grade Level 
Student standard on each EOC assessment were further disaggregated by student demographic and 
educational variables. The demographic and educational composition of students in the sample was 
compared by LE treatment and non-LE groups. Multiple regressions were conducted to identify key 
demographic and educational predictors and LE on STAAR EOC performance for students in the sample. 
The unstandardized coefficients and standardized coefficient (Beta) were presented. Beta is used to 
compare the strength and direction of predictors. 


Limitations 


e It is likely that teachers in this sample may have had multiple exposures to the Literacy Empowered 
PD. Control for multiple exposures was not considered in the analyses, but these might have impacted 
students’ performance as they represented additional doses of PD. 


e Literacy Empowered is a literacy strategy and yet the outcomes used in this evaluation constituted 
other academic disciplines besides reading and English Language Arts (ELA). It is likely that there may 
have been limited correlations between the content and strategies of Literacy Empowered and some 
of the content areas. It is believed, however, that literacy impacts other academic disciplines and should 
have positive effects on the results of non-ELA EOC assessments. 


e HISD is undertaking several literacy initiatives concurrently and is implementing other programs to 
enhance students’ performance. These programs were not controlled for in this study, which could have 
contaminated the study. To minimize the effects, only students whose teachers were exposed to the 
Literacy Empowered PD were included in the treatment group. Further, treatment effects with 
regression adjustment estimators were used to minimize the contamination effects. 


Results 
What was the PD participation for Literacy Empowered among teachers in the study? 


Figure 1 shows the distribution of teachers who participated in the LE professional development by credit 
hours completed prior to its implementation in the classroom during the 2018-2019 school year. Details 
are in Table A1 (Appendix A, p. 14). 


Figure 1. Percentage Distribution of Teachers Participation in LE Professional Development, 2018- 
2019 
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e Literacy Empowered PD participants completed between three and twenty-one credit hours. 


e Most teachers (53.2%) in the study completed six credit hours and 78.7 percent completed either three 
or six credits. 


e About 13.8 percent of teachers in the study completed 18 credit hours and 7.4 percent had the largest 
number of credits (21 hours). 


What was the demographic and educational composition of students in the 2018-2019 LE study 
sample? 


Figure 2 shows the percentage distribution of students who made up the study sample. It displays the 
demographic and educational composition of students whose teachers completed professional 
development (PD) prior to the delivery of LE and students whose teachers did not participate in the PD. 
Details are provided in Table A2 (Appendix A, p. 14). 


Figure 2. Demographic and Educational Composition of the Literacy Empowered Study Sample, 


2018-2019 
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o 
o 50.0 
oO 
a 
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30.0 
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‘so | , 
0.0 Ba Ll. all Bu 
Fem. | Male |Asian | Black | Hisp. |White} No | Yes | No | Yes | No | Yes | No | Yes |Span.} Eng. | Other} No | Yes 
Gender Ethnicity Econ. At Risk Special Ed LEP Home Lang. G/T 
Disadv. 
mNon-LE} 51.5 | 48.5 | 4.1 | 22.6 | 61.9 | 10.0 | 24.7 | 75.3 | 43.5 | 56.5 | 93.9 | 6.1 | 82.7 | 17.3 | 43.3 | 52.1] 4.6 | 80.9 | 19.1 
BLE 48.9 | 51.1 | 1.8 | 28.9 | 65.1 | 3.7 | 14.2 | 85.8 | 28.6 | 71.4 | 91.6 | 8.4 | 79.0 | 21.0 | 47.9 | 49.6] 2.5 | 92.6] 7.4 


Source: Chancery Ad Hoc using Cognos, downloaded, 08/19/19 (data only). 
Note: Fem.= female; Hisp. = Hispanic, Econ. Disadv. = economically disadvantaged; Special Ed = special education; LEP = 
limited English proficiency; Home Lang = home language; G/T= gifted and talented. 


e Ahigher percentage of male (51.1 vs. 48.5%) students, Black (28.9 vs. 22.6%) and Hispanic (65.1 vs. 
61.9%) students make up the LE treatment group compared to the non-treatment group. There were 
fewer Asian (1.8 vs. 4.1%) and White (3.7 vs. 10.0%) students in the treatment group than in the non- 
treatment group. 


e There were higher proportions of economically-disadvantaged (85.8 vs. 75.3%), more at risk (71.4 vs. 
56.5%), and more special education (8.4 vs. 6.1%) students in the treatment group compared to similar 


groups of students in the non-treatment group. 


e Based on home languages, the highest proportion of students from the LE treatment group came from 
homes where English (49.6%) was the predominant language spoken. However, there was a higher 
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proportion of students from predominantly Spanish-speaking homes (47.9%) in the treatment group 
compared to students from similar homes (43.3%) in the non-treatment group. 


e A substantially lower percentage of G/T students (7.4%) comprised the treatment group compared to 
G/T students (19.1%) in the non-treatment group. 


How did students exposed to LE perform on the 2019 State of Texas Assessments of Academic 
Readiness (STAAR) End-of-Course (EOC)? 


Table B1 through Table B5 (Appendix B, pp.15—16) show the effects of Literacy Empowered on the 
students whose teachers completed the LE professional development. As mentioned, the tables show how 
a student selected at random from the sample would have scored (Average Treatment Effect (ATE)), that 
is, the difference between the potential outcome means (POM). 


e Astudent selected at random from the sample and exposed to LE would have scored less well on the 
2019 STAAR Algebra | EOC test compared to a student who was not exposed to the same LE. On 
average, the scale score of a student selected at random from the sample would have been statistically 
significant but lower (-41.6 ssp) than the Algebra | EOC POM (4000.3 ssp) for students who were not 
exposed to LE during the 2018-2019 school year (p<.05, two-tailed). Details are in Table B1 (Appendix 
B, p. 15). 


e Astudent selected at random from the sample and exposed to Literacy Empowered would have scored 
just as well on the 2019 STAAR Biology EOC test compared to a student who was not exposed to the 
same LE. On average, the scale score of a student selected at random from the sample would not have 
been statistically different (-27.6 ssp) from the Biology EOC POM (4151.5 ssp) for students who were 
not exposed to LE during the 2018-2019 school year (p>.001, two-tailed). Details are in Table B2 
(Appendix B, p. 15). 


e Astudent selected at random from the sample and exposed to LE would have scored less well on the 
2019 STAAR English | test compared to a student who was not exposed to the same LE. On average, 
the scale score of a student selected at random from the sample could have been statistically significant 
but lower (-66.0 ssp) than the English | EOC POM (4048.2 ssp) for students who were not exposed to 
LE during the 2018-2019 school year (p<.05, two-tailed). Details are in Table B3 (Appendix B, p. 15). 


e Astudent selected at random from the sample and exposed to LE would have scored less well on the 
2019 STAAR English II test compared to a student who was not exposed to the same LE. On average, 
the scale score of a student selected at random from the sample would have been statistically 
significant but lower (-87.7ssp) than the English Il EOC POM (4104.3 ssp) of students who were not 
exposed to LE during the 2018-2019 school year (p<.001, two-tailed). Details are in Table B4 
(Appendix B, p. 16). 


e Astudent selected at random from the sample and exposed to LE would have scored less well on the 
2019 STAAR U.S. History EOC test. On average, the scale score of a student selected at random from 
the sample would have been statistically significant but lower (-92.5 ssp) than the U.S. History POM 
(4307.8 ssp) for students who were not exposed to LE during the 2018-2019 school year. Details are 
in Table B5 (Appendix B, p. 16). 


Figure 3 shows the comparative percentage of students who met the Approaches, Meets, and Masters 
Grade Level Student standards on the 2019 STAAR EOC tests. Students in the treatment and non- 
treatment groups were compared. Details are in Table B6 (Appendix B, p.16). 
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Figure 3. Percentage of LE and Non-LE Students who Met Grade Level Student Standards on the 
2019 STAAR EOC Tests 
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Source: Chancery Ad Hoc using Cognos, downloaded, 08/19/19 (data only). 


e A higher percentage of students in the non-treatment (non-LE) group compared to students in the 
treatment (LE) group met the Approaches, Meets, and Masters Grade Level Student standard on the 
five 2019 STAAR EOC tests. 


e English | and English Il EOC tests had the largest performance gaps (21.1 and 11.3 percentage points, 
respectively) between the treatment and non-treatment groups of students who performed at or above 
the Approaches Grade Level Student standard, in favor of the non-treatment or non-LE group. U.S 
History had the smallest performance gap of 2.1 percentage points. 


Table B7 (Appendix B, p.17) shows the percentage of students in the study disaggregated by key 
demographic and educational variables who performed at or above the Approaches Grade Level Student 
standard on the 2019 STAAR EOC tests. 


e Generally, a higher percentage of students in the non-treatment subgroups, with some exceptions, 
performed at or above the Approaches Grade Level Student standard on the 2019 STAAR EOC tests. 


e A higher percentage of special education students in the treatment (LE) group compared to similar 
students in the non-treatment group performed at or above the Approaches Grade Level Student 


standard on four of the five 2019 STAAR EOC tests. 


e A higher percentage of Asians in the treatment group compared to their peers in the non-treatment 
group performed at or above the Approaches Grade Level Student standard in Algebra | and Biology. 
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e = Slightly higher proportions of non-economically disadvantaged (Algebra |), Black (U.S. History), non- 
at-risk (Biology), at-risk (U.S. History), G/T (U.S. History) students, and White and students with 
predominant home languages other than Spanish and English (English Il) in the treatment group 
performed at or above the Approaches Grade Level Student standard compared to their peers in the 
non-treatment group. 


To what extent did LE contribute to the 2018-2019 reading achievement of high school students in 
the sample? 


Table C1 to Table C5 (Appendix C, pp. 18-20) provides details on the selected predictors of performance 
on the 2019 STAAR Algebra I, Biology, English |, English Il, and U.S. History EOC exams for students in 
the study sample. As mentioned earlier, twelve key demographic and educational factors including LE 
participation were included in the model. A single model was used. 


Algebra | 


e Overall, the model accounted for 21.4 percent of the variance in the Algebra | EOC exam performance 
of students in the sample. The constant or mean of 4234.5 ssp was statistically significant (p<.001). 


e Eight of the twelve variables in the model were statistically significant (p<.001; p.<.05) predictors of the 
2019 Algebra | EOC exam performance of students in the sample. Being identified as being at risk for 
school dropout (at-risk) (30%); special education (19%), and Gifted and Talented (17%) were the 
strongest predictors. At-risk and special education were adverse predictors. 


e Literacy Empowered predicted 2.0 percent of the variance of the Algebra | EOC exam performance of 
students in the study sample. The variance was statistically significant but inverse (p<.05). Details are 
in Table C1 (Appendix C, p. 18). 


Biology 
e Overall, the Biology EOC exam regression model accounted for 45.8 percent of the variance in the 
performance of students in the sample with a constant or mean of 4451.3 ssp. 


e Ten of the twelve variables in the model were statistically significant (p<.001; p<.05) predictors of the 
2019 STAAR Biology EOC exam performance of students in the study sample. Being at risk (36%), 
G/T (30%), and being Hispanic (14%) were the strongest predictors of performance for Biology. G/T 
was a positive predictor of performance. Details are in Table C2 (Appendix C, p. 18) 


e Literacy Empowered accounted for just one percent of the variance in the Biology EOC Exam 
performance of students in the study sample. The variance was not statistically significant (p>.001). 


English | 


e The regression model explained 48.2% of the variance in performance on the 2019 STAAR English | 
EOC test. The statistically significant (p<.001) constant or mean was 4423.6 ssp. 


e Nine of the twelve variables in the model were statistically significant (p<.001; p<.05) predictors of the 
2019 English | EOC exam performance of students in the study sample. At-risk (38%), G/T (27%), and 
special education (17%) were the strongest predictors. G/T was a statistically significant (p<.001) 
positive predictor. Details are in Table C3 (Appendix C, p. 19) 
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e Literacy Empowered (3.0%) was a statistically significant but an inverse predictor of the English | EOC 
exam performance of students in the sample (p<.001). 


English Il 


e The regression model accounted for 48.0 percent of the variance in the 2019 STAAR English II EOC 
exam performance of students in the sample. The constant or mean of 4417.4 ssp was statistically 
significant (p<.001). 


e Nine of the twelve variables in the model were statistically significant predictors of the 2019 English II 
EOC exam performance of students in the sample. At risk, (40%), G/T (29%), and special education 
(16%) were the strongest predictors. G/T was a statistically significant (p<.001) positive predictor. 
Details are in Table C4 (Appendix C, p. 19). 


e LE (3.0%) was a statistically significant but inverse predictor of the English Il EOC exam performance 
of students in the study sample (p<.001). 


U.S. History 


e The regression model accounted for 36.2 percent of the variance in the 2019 U.S. History EOC exam 
performance of students in the sample. The constant or mean of 4557.4 was statistically significant 
(p<.001). 


e Tenof the twelve variables in the model were statistically significant predictors of the 2019 STAAR U.S. 
History EOC exam performance of students in the study sample. At-risk (30%), G/T (28%), and special 
education (13%) were the strongest predictors. G/T was a statistically significant (p<.001) positive 
predictor. Details are in Table C5 (Appendix C, p. 20). 


e Literacy Empowered (2.0%) was a statistically significant (p<.001) inverse predictor of the 2019 STAAR 
U.S. History EOC exam performance for students in the sample. 


Discussion 


The purpose of this evaluation was to measure the effect of Literacy Empowered on the academic 
performance of students whose teachers completed the PD in preparation for delivering the initiative during 
the 2018-2019 school year. The study used treatment effects with regression adjustment (teffects ra) ona 
sample of students whose teachers completed the PD and those whose teachers did not complete the PD. 
Teffects ra estimated the effects of the initiative on the 2019 STAAR Algebra |, Biology, English |, English 
ll, and U.S. History EOC test performance of students by calculating the potential outcome means (POM) 
for each group and used the differences between these two means as the effects. 


Findings indicated that students whose teachers did not participate in the LE professional development 
outperformed students on the 2019 STAAR Algebra |, English |, English Il, and U.S. History EOC exams 
whose teachers completed the LE professional development. The difference for Biology was not statistically 
significant. A higher percentage of students whose teachers did not participate in the professional 
development compared to the students whose teachers did participate in the professional development 
performed at or above the Approaches Student Standard on the five STAAR EOC exams in the study. 
When disaggregated by key demographic and educational student groups, the trend was similar in that a 
higher percentage of student groups whose teachers did not participate in the professional development 
met the Approaches Student Standard except for special education students and several other student 
groups in the various content areas. 
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Compared to their peers whose teachers did not participate in the LE professional development, a higher 
percentage of special education students whose teachers completed the PD met the Approaches Grade 
Level Student standard on four of the five STAAR EOC exams in this study. Asian students in the treatment 
group outperformed their peers in the non-treatment group in two STAAR EOC subjects. Other subgroups 
(Black, non-economically disadvantaged, non-at risk, at-risk, G/T students, and students with predominant 
home languages other than Spanish or English) outperformed their peers in the non-treatment group on 
one EOC exam. The limited number of professional development hours, the extent to which the LE initiative 
was delivered with fidelity, and the possible effect of non-program variables, like students’ at-risk statuses, 
may point to the differences in student performance. However, the slightly higher performance of students 
in selected treatment subgroups does indicate that the program has the potential to be effective if 
implemented with fidelity including yearlong teacher support. 


The study showed that most teachers (78.7%) in the study did not complete the number of professional 
development hours identified in the literature as essential to substantially improve student achievement. 
Teachers were not exposed to yearlong support as suggested in the literature on effective professional 
development. Professional development effectiveness research suggests that students increase their 
achievement by 21 percentile points when their teachers were exposed to substantial professional 
development hours (49 hours). 


Being identified as at-risk in the sample was the strongest but an inverse predictor of performance on the 
STAAR EOC tests. Addressing issues related to at-risk students may be essential to improve the 
performance of LE students, overall. In addition, implementing professional development with fidelity and 
ensuing that yearlong support is available to both monitor implementation and respond to related questions, 
queries, and issues are essential for the effectiveness of LE. G/T, which can be considered a proxy for 
previous performance, was a positive predictor of performance on the STAAR EOC tests. A further 
indication that students benefited more from their previous knowledge and abilities than exposure to LE. 
LE explained small but inverse variation in students’ scores on the STAAR EOC tests. LE contained most, 
if not all, of the key strategies the research identified as being effective in high school literacy instruction. 
Consistency in delivering these strategies is essential for program fidelity and to improve student 
performance. Greater attention needs to be paid to those areas that are inverse predictors of student 
performance on the STAAR EOC tests. 
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Appendix A 
Table A1. Distribution of Teachers by Credit Hours Who Completed the Literacy Empowered 
PD, 2018-2019 
Credit Hour N % Total Credit Hours 
3 24 25.5 72 
6 50 53.2 300 
18 13 13.8 234 
21 {4 7.4 147 
Total 94 753 


Source: 2018-2019 e-TRAIN: Employee Training, HISD Research and Accountability Archival Database 


Table A2. Demographic and Educational Composition of Students in the Study Sample, 2018- 


2019 
Study Sample 


Variables Non-Literacy Empowered Literacy Empowered 
n % n % 
Female 25,917 51.5 1,627 48.9 
Gender 
Male 24,364 48.5 1,702 51.1 
Asian 2,053 4.1 60 1.8 
ma Black 11,361 22.6 961 28.9 
Ethnicity ; : 
Hispanic 31,112 61.9 2,168 65.1 
White 5,007 10.0 123 3.7 
: No 12,404 24.7 473 14.2 
Econ. Disadv. 
Yes 37,877 75.3 2,856 85.8 
N 21,897 i 952 ‘ 
Apis fe) ‘ 43.5 28.6 
Yes 28,384 56.5 2,377 71.4 
: No 47,193 93.9 3,050 91.6 
Special Ed. 
Yes 3,088 6.1 279 8.4 
= No 41,560 82.7 2,630 79.0 
Yes 8,721 17.3 699 21.0 
Spanish 21,794 43.3 1,596 47.9 
Home Language English 26,183 52.1 1,651 49.6 
Other 2,304 4.6 82 2.5 
N 40,683 : 3,083 . 
Gifted and Talented z : ue eae 
Yes 9,598 19.1 246 7.4 
Non CTE 11,483 22.8 587 17.6 
Career& Technical 
Education (CTE) Non-Coherent 12,406 24.7 612 18.4 
Coherent 26,392 52.5 2,130 64.0 


Source: Chancery Ad Hoc using Cognos, downloaded, 08/19/19 (data only). 
Note: Econ. Disadv. = economically disadvantaged; Special Ed = special education; LEP -= limited English proficiency. 
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Appendix B 


Table B1. Treatment Effects of LE on the 2019 STAAR Algebra | End-of-Course Student 
Performance 


Scale Score Robust 
Coefficient Standard z P>z ade 
n = 8,466 Error 
Average Treatment Effect 
LE 

(1 vs 0) -41.6 20.9 -1.99 0.046 [-82.49, -0.68] 
Potential Outcome Mean 

Non-LE 

0 4000.3 5.7 700.94 0.000 (3989.08, 4011.45] 


Table B2. Treatment Effects of LE on the 2019 STAAR Biology End-of-Course Student 


Performance 
Scale Score Robust " 
Coefficient Standard z P>z Bee aie 
n= 11,965 Error 
Average Treatment Effect 
LE 
(1 vs 0) -27.6 20.2 -1.36 0.173 [-67.29, 12.07] 
Potential Outcome Mean 
Non-LE 
(6) 4151.5 5.2 804.9 0.000 [4141.44, 4161.66] 


Table B3. Treatment Effects of LE on the 2019 STAAR English | End-of-Course Student 


Performance 
Scale Score Robust 
Coefficient Standard z P>z [95% Conf. Interval} 
n = 10,836 Error 
Average Treatment Effect 
LE 

(1 vs 0) -66.0 23.3 -2.83 0.005 [-111.65, -20.36] 
Potential Outcome Mean 

Non-LE 

0 4084.2 5.7 714.07 0.000 [4073.02, 4095.44] 
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Table B4. Treatment Effects of LE on the 2019 STAAR English II End-of-Course Student 


Performance 


Scale Score Robust 
Coefficient Standard z P>z [95% Conf. Interval} 
n= 11,208 Error 
Average Treatment Effect 
LE 

(1 vs 0) -87.7 15.4 -5.69 0.000 [-117.94, -57.50] 
Potential Outcome Mean 

Non-LE 

0 4104.3 5.5 748 0.000 [4093.56, 4115.07] 


Table B5. Treatment Effects of LE on the 2019 STAAR U.S. 


History End-of-Course Student 


Performance 
Scale Score Robust 
Coefficient Standard z P>z [95% Conf. Interval} 
n= 11,135 Error 
Average Treatment Effect 
LE 

(1 vs 0) -92.5 19.6 -4.72 0.000 [-130.87, -54.05] 
Potential Outcome Mean 

Non-LE 

0 4307.8 5.3 805.74 0.000 [4297.37, 4318.33] 


Table B6. Comparative Percentage of LE and Non-LE Students who Met Grade Level Student 


Standards on the 2019 STAAR EOC Tests 


Approaches Meets Masters 
STAAR 
EOC Non-LE LE Non-LE LE Non-LE LE 
n % n % n % n % n % n % 
Algebra | 6,543 | 82.5 | 408 76.8 3,951 | 49.8 | 208 39.2 1,949 | 24.6 94 | 17.7 
Biology 9,981 | 88.4 | 547 80.6 6,745 | 59.8 | 322 47.4 2,530 | 22.4 | 100 | 14.7 
English | 7,757 | 75.2 | 285 54.1 6,215 | 60.3 | 189 35.9 1,644 | 15.9 33 | 6.3 
English II 8,006 | 77.2 | 555 65.9 6,352 | 61.3 | 380 45.1 1,126 | 10.9 15 | 18 
U.S. History | 9,658 | 93.0 | 682 90.9 7,931 | 72.5 | 455 60.7 4,476 | 43.1 | 200 | 26.7 


Source: Chancery Ad Hoc using Cognos, downloaded, 08/19/19 (data only). STAAR Regular, first-time testers, paper and online administration mode. 


Note: *p<.005; **p.001 


Approaches (scale scores): Algebra 1 = 3500-3961; Biology = 3500-3966; English | = 3750-3965; English Il = 3750-3966; U.S. History = 3500-— 


3980. 


Meets (scale scores): Algebra | = 4000-4288; Biology = 400-4495; English |: 4000-4603; English Il = 4000-4730; U.S. History = 4000-4375 
Masters (Scale scores): Algebra 14333-6181; Biology = 4576-6229; English | = 4691-6367; English Il = 4831-6416; U.S. History = 4440-6609 
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Table B7. Disaggregated Comparative Percentage of Students in the Study Sample who Performed At or 


Level Student Standard on the 2019 STAAR EOC Tests 


Approaches Grade 


Non-LE Students LE Students 
Variable Algebra | Biology English | English II U.S. History Algebra | Biology English | English II U.S. History 
n % n % n % n % n % n % n % n % n % n % 
Female 3,513 86.4 5,219 90.7 4,307 80.7 4,441 82.1 5,026 94.0 206 78.9 285 84.6 154 60.9 295 72.8 338 91.1 
arts Male 3,030 78.3 4,762 86.0 3,450 69.4 3,565 72.0 4,632 92.0 202 74.8 262 76.6 131 47.8 260 59.5 344 90.8 
Asian 139 93.3 447 97.4 446 90.8 437 92.2 456 95.0 10 100.0 14 100.0 13 86.7 10 90.9 9 90.0 
7 Blacks 1,720 82.5 2,199 87.4 1,607 71.5 1,682 74.6 2,088 92.6 132 82.0 173 83.6 92 54.4 139 62.3 188 93.5 
Sree Hisp. 4,102 81.9 6,106 86.8 4,539 72.2 4,744 74.3 5,891 92.1 250 73.3 331 78.4 161 50.8 379 65.5 458 90.0 
White 493 84.7 1,050 96.1 1,013 91.7 1,009 91.1 1,090 97.4 14 82.4 26 81.3 16 69.6 25 96.2 22 88.0 
Econ. No 1,300 87.2 2,615 95.4 2,412 89.9 2,467 90.6 2,663 96.3 64 87.7 89 87.3 58 73.4 76 78.4 112 91.8 
Disadv. Yes 5,243 81.4 7,366 86.2 5,345 70.1 5,539 72.5 6,995 91.8 344 75.1 458 79.4 227 50.7 479 64.3 570 90.8 
; No 2,359 95.2 4,784 98.5 4,549 96.5 4,798 97.6 4,916 99.6 109 92.4 194 99.0 121 89.6 261 95.3 227 99.1 
ee Yes 4,184 76.7 5,197 80.8 3,208 57.4 3,208 58.8 4,742 87.0 299 72.4 353 73.1 164 41.8 294 51.8 455 87.3 
No 5,199 85.8 8,519 93.2 7,215 84.8 7,586 85.3 8,609 96.1 326 82.1 459 87.6 267 66.3 532 75.8 575 95.2 
a Yes 1,344 71.6 1,462 68.0 542 30.2 420 28.5 1,049 73.7 82 61.2 88 56.8 18 14.5 23 16.4 107 73.3 
Special No 6,244 85.3 9,574 90.4 7,633 78.5 7,838 80.3 9,280 94.7 378 78.9 511 82.6 274 57.4 541 69.4 643 92.4 
edleation Yes 299 48.5 407 | 58.7 | 124 | 213 | 168 | 27.5 | 378 | 646 | 30 57.7 | 36 | 60.0 | 11 | 220 | 14 | 222 | 39 | 722 
No 5,885 81.0 7,727 85.7 5,464 68.6 5,788 71.3 7,589 91.3 394 76.5 492 79.1 258 51.7 469 62.6 630 90.3 
ca Yes 658 98.7 2,254 99.3 2,293 97.9 2,218 98.7 2,069 99.9 14 87.5 55 96.5 27 96.4 86 92.5 52 100.0 
Spanish 2,924 81.5 4,142 84.2 2,942 67.1 3,101 70.0 4,067 91.0 181 71.5 233 75.2 104 44.4 254 62.1 351 90.0 
as English 3,360 83.3 5,391 91.9 4,431 81.9 4,553 83.2 5,136 95.0 218 82.0 302 85.6 173 62.7 285 68.7 318 93.3 
Other 259 83.0 448 89.6 384 74.4 352 75.5 455 89.2 9 75.0 12 75.0 8 47.1 16 88.9 13 68.4 
Source: Chancery Ad Hoc using Cognos, downloaded, 08/19/19 (data only). STAAR Regular, First-time testers, paper and online administration modes. 
Note: Hisp. = Hispanic; Approaches Student Standard (scale scores): Algebra 1 = 3500-3961; Biology = 3500-3966; English | = 3750-3965; English Il = 3750-3966; U.S. History = 3500-3980 
Shaded blue = Students in the LE treatment group who outperformed their peers in the non-treatment group 
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Appendix C 

Table C1. Selected Predictors of the 2019 STAAR Algebra | EOC Performance for Students in the 

LE Sample 
Algebra | Unstandardized Standardized : 
Scale Score Coefficient Beta [95% Confidence Interval] 
Econ. Disadv. 13.8 0.01 [-13.4, 41.1] 
At-Risk -330.1** -0.30 [-352.6, -307.6] 
Special Education -354.5** -0.19 [-390.8, -318.1] 
CTE 36.6** 0.06 [24.4, 48.8] 
GIT 320.8** 0.17 [283.8, 357.8] 
American Indian/Alaskan Native -37.1 -0.03 [-93.3, 19.1] 
Asian 295.0** 0.08 [210.0, 379.9] 
African American -79.1* -0.07 [-142.4, -15.8] 
White -58.3* -0.06 [-114.1, -2.6] 
Hawaiian Native/Pacific Islanders -134.6 -0.02 [-271.5, 2.3] 
Hispanic -14.1 -0.01 [-51.1, 22.8] 
Literacy Empowered -51.6* -0.02 [-91.6, -11.6] 
Constant 4234.5** (4167.7, 4301.4] 
F 192.7** 
R? 21.4 
N 8,466 


Source: Chancery Ad Hoc using Cognos, downloaded, 08/19/19 (data only). 
*p<.005; **p.001 


Table C2. Selected Predictors of the 2019 STAAR Biology EOC Performance for Students in the 


LE Sample 
ae unstandayized Stangardized (95% confidence interval 
Econ. Disadv. -98.2** -0.07 [-118.0, -78.3] 
At-Risk -411.5** -0.36 [-428.3, -394.7] 
Special Education -303.6** -0.13 [-334.5, -272.7] 
CTE 22.0** 0.03 [12.7, 31.2] 
G/T 416.5** 0.30 [395.7, 437.2] 
American Indian/Alaskan Native 53.9* 0.04 [12.6, 95.1] 
Asian 238.6** 0.09 [185.5, 291.7] 
African American -106.0** -0.08 [-151.8, -60.3] 
White 48.6* 0.04 [8.0, 89.2] 
Hawaiian Native/Pacific Islanders 28.6 0.00 [-78.5, 135.6] 
Hispanic -159.6** -0.14 [-185.8, -133.3] 
Literacy Empowered -26.6 -0.01 [-58.6, 5.4] 
Constant 4451.3** [4403.2, 4499.4] 
F 843.19** 
R2 45.8 
N 11,965 


Source: Chancery Ad Hoc using Cognos, downloaded, 08/19/19 (data only). 
*p<.005; **p.001 
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Table C3. Selected Predictors of the 2019 STAAR English | EOC Performance for Students in the 


LE Sample 
English | Unstandardized Standardized 5 . 
Scale Score Coefficient Beta [paar Confidence Interval 
Econ. Disadv. -128.6** -0.10 [-149.9, -107.4] 
At-Risk -448.9** -0.38 [-467.1, -430.7] 
Special Education -435.2** -0.17 [-469.9, -400.5] 
CTE -16.0* -0.02 [-25.9, -6.1] 
G/T 386.7 0.27 [364.9, 408.4] 
American Indian/Alaskan Native 96.9** 0.06 [51.0, 142.9] 
Asian 174.6** 0.07 [118.5, 230.7] 
African American -40.6 -0.03 [-90.6, 9.4] 
White 70.0* 0.06 [25.1, 115.0] 
Hawaiian Native/Pacific Islanders 11.0 0.00 [-104.4, 126.3] 
Hispanic -148.4** -0.12 [-176.1, -120.7] 
Literacy Empowered -90.5** -0.03 [-127.9, -53.2] 
Constant 4423.6** [4371.2, 4476.0] 
F 842.7** 
R? 48.2 
N 10,836 


Source: Chancery Ad Hoc using Cognos, downloaded, 08/19/19 (data only). 
*p<.005; **p.001 


Table C4. Selected Predictors of the 2019 STAAR English Il EOC Performance for Students in the 


LE Sample 


English Il Standardized Standardized 5 : 

Scale Score Coefficient Beta [pa Confidence Interval 
Econ. Disadv. -112.1** -0.09 [-132.0, -92.1] 
At-Risk -449.8** -0.40 [-466.6, -433.1] 
Special Education -393.2** -.0.16 [-425.7, -360.6] 
CTE -7.6 -0.01 [-17.3, 2.2] 
G/T 403.5** 0.29 [383.0, 424.1] 
American Indian/Alaskan Native 41.8 0.03 [-1.1, 84.7] 
Asian 161.8** 0.06 [109.6, 214.0] 
African American -39.1 -0.03 [-86.3, 8.0] 
White 49.8* 0.04 [7.6, 92.0] 
Hawaiian Native/Pacific Islanders -156.6* -0.02 [-275.5, -37.7] 
Hispanic -115.7** -0.10 [-141.9, -89.5] 
Literacy Empowered -61.7** -0.03 [-90.7, -32.7] 
Constant 4417.4** [4368.4, 4466.4] 
F 863.0 

R?2 48.0 

N 11,208 


Source: Chancery Ad Hoc using Cognos, downloaded, 08/19/19 (data only). 
*p<.005; **p.001 
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Table C5. Selected Predictors of the 2019 STAAR U.S. History EOC Performance for Students in 


the LE Sample 


2 unstandarized Standardized (a5 confidence interval 
Econ. Disadv. -110.0** -0.09 [-130.8, -89.1] 
At-Risk -334.0** -.030 [-352.1, -315.9] 
Special Education -302.1** -0.13 [-337.9, -266.2] 
CTE -8.2 -0.01 [-18.6, 2.2] 
G/T 389.3** 0.28 [366.4, 412.1] 
American Indian/Alaskan Native 73.5* 0.06 [24.0, 123.1] 
Asian 140.4** 0.06 [80.6, 200.1] 
African American -83.7* -0.06 [-137.9, -29.4] 
White 90.5** 0.08 [41.7, 139.4] 
Hawaiian Native/Pacific Islanders 61.6 0.01 [-80.2, 203.4] 
Hispanic -137.5** -0.12 [-165.6, -109.4] 
Literacy Empowered -53.2** -0.02 -86.0, -20.4 
Constant 4557.4** 4501.1, 4613.8 
F 528.3 

R? 36.2 

N 11,135 
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